2 links tagged with all of: superintelligence + alignment + ai safety
Click any tag below to further narrow down your results
Links
Sean Goedecke argues that LLMs don't have fixed goals and their internal workings can't yield the kind of motive-level understanding needed to guarantee safety, so "solving alignment" for them is incoherent. He concludes that mechanistic interpretability, though useful, can't deliver the confidence required to prevent rogue superintelligent behavior.
- LLMs are trained to imitate human text, so they lack a single fixed goal that could be formally specified, unlike chess engines or the classic "blind optimizer" paperclip model.
- Mechanistic interpretability can trace calculations but not motive, and it can only label behaviors we've already elicited, leaving many failure modes unobserved.
- Feature labels are guesses that can't be distinguished from alternatives (the "kindness unless 2030" problem), and the combinatorial complexity per token is too large for humans to check.
- Current mitigations like chain-of-thought monitoring and sandboxing are useful but judged insufficient against a superintelligence.
AI researchers have spent two decades seriously arguing that superintelligent AI could kill everyone, and this isn't a PR stunt — it's a genuine belief that shapes how they work. The author explains what they think could happen and why they keep building AI anyway.
- AI researchers use the term "p(doom)" to discuss extinction probability and have been publishing on this since at least 2008, making it an established part of AI safety culture, not a recent panic.
- Multiple plausible kill mechanisms exist: engineered bioweapons, triggering nuclear war through military AI systems, robot takeover, or unforeseen methods from a superintelligence we can't predict.
- The "race dynamics" problem: if superintelligent AI is inevitable, being first might be the only way to ensure it's aligned with human values, which creates a perverse incentive to build faster rather than safer.